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Abstract — NASA’s Goddard Space Flight Center has developed a 
fast and effective method for generating image segmentation 
hierarchies. These segmentation hierarchies organize image data 
in a manner that makes their information content more 
accessible for analysis. Image segmentation enables analysis 
through the examination of image regions rather than individual 
image pixels. In addition, the segmentation hierarchy provides 
additional analysis clues through the tracing of the behavior of 
image region characteristics at several levels of segmentation 
detail. The potential for extracting the information content from 
imagery data based on segmentation hierarchies has not been 
fully explored for the benefit of the Earth and space science 
communities. This paper explores the potential of exploiting these 
segmentation hierarchies for the analysis of multi-date data sets, 
and for the particular application of change monitoring. 
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I. Introduction 

Image segmentation is the partitioning of an image into 
related sections or regions. For remotely sensed images of the 
earth, an example of an image segmentation would be a labeled 
map that divides the image into areas covered by distinct earth 
surface covers such as water, snow, types of natural vegetation, 
types of rock formations, types of agricultural crops and types 
of other man created development. In unsupervised image 
segmentation, the labeled map may consist of generic labels 
such as region 1, region 2, etc., which may be converted to 
meaningful labels by a post-segmentation analysis. 

Segmentation is a key first step for a number of approaches 
to image analysis and compression. In image analysis, the 
group of image points contained in each region provides a good 
statistical sampling of image values for more reliable labeling 
based on region mean feature values. In addition, the region 
shape can be analyzed for additional clues to the appropriate 
labeling of the region. In image compression, the regions form 
a basis for compact representation of the image. The quality of 
the prerequisite image segmentation is a key factor in 
determining the level of performance for these image analysis 
and compression approaches. 

A segmentation hierarchy is a set of several segmentations 
of the same image at different levels of detail in which the 
segmentations at coarser levels of detail can be produced from 


simple merges of regions at finer levels of detail. This is useful 
for applications that require different levels of image 
segmentation detail depending on the particular image objects 
segmented. A unique feature of a segmentation hierarchy that 
distinguishes it from most other multilevel representations is 
that the segment or region boundaries are maintained at the full 
image spatial resolution for all levels of the segmentation 
hierarchy. 

In a segmentation hierarchy, an object of interest may be 
represented by multiple image segments in finer levels of detail 
in the segmentation hierarchy, and may be merged into a 
surrounding region at coarser levels of detail in the 
segmentation hierarchy. If the segmentation hierarchy has 
sufficient resolution, the object of interest will be represented 
as a single region segment at some intermediate level of 
segmentation detail. The segmentation hierarchy may be 
analyzed to identify the hierarchical level at which the object of 
interest is represented by a single region segment. The object 
may then be identified through its spectral and spatial 
characteristics. Additional clues for object identification may 
be obtained from the behavior of the image segmentations at 
the hierarchical segmentation levels above and below the level 
at which the object of interest is represented by a single region. 

Segmentation hierarchies may be formed through a region 
growing approach to image segmentation. In region growing, 
spatially adjacent regions iteratively merge through a specified 
merge selection process. Hierarchical Step-Wise Optimization 
(HSWO) is a form of region growing segmentation in which 
the iterations consist of finding the best segmentation with one 
region less than the current segmentation [1], [2]. The best 
segmentation is defined through a mathematical criterion such 
as a minimum vector norm or minimum mean squared error. 
HSWO may be initialized from a pre-segmentation provided by 
another segmentation approach, or may be initialized from 
single pixel regions. An augmentation of HSWO, called HSEG 
(for Hierarchical Segmentation), was introduced in [3] in which 
spatially non-adjacent regions are allowed to merge controlled 
by a threshold based on previous merges of spatially adjacent 
regions. HSEG also includes a method for selecting the most 
“significant” iterations from which the segmentation result is 
saved into the output segmentation hierarchy. A recursive, 
divide-and-conquer, approximation of HSEG, called RHSEG, 
was first introduced in [4] and further described in [5], A 
parallel implementation of RHSEG is given in [6], The latest 



refinements of HSEG and RHSEG are detailed in [7] and [8], 
The current implementation of RHSEG is so efficient that a lull 
Landsat Thematic Mapper (TM) scene (roughly 7000 by 6500 
pixels) can be processed in 5 to 10 minutes (depending on 
parameter settings) on a Beowulf cluster consisting of 256 
2.4GHz CPUs ( http ://thun derhead. gsfc . nasa. gov ). This is only 
10 to 20 times the amount of time the Landsat TM sensor takes 
to collect this amount of data. 

The current state of change detection/environmental 
monitoring tends to use data sets collected with either high 
temporal frequency and moderate spatial resolution, or limited 
observations at high spatial resolution. This is, for the most 
part, a simple artifact of the revisit time of the sensors or the 
spatial resolution required for detection of the phenomenon 
under study. A good example of the former is NOAA NESDIS 
Automated Biomass Burning Algorithm [ABBA 
http://www.ssd.noaa.gov/PS/FIRE/hms.html] where wild fires 
detected with the NOAA Advanced Very High Resolution 
Radiometer [AVHRR] and posted in near-real-time to their 
Web site for the user community. An example of the latter 
approach would be a monitoring program like the Land Cover 
Mapping and Monitoring Program (LCMMP) of the USDA 
Forest Service (FS) and the California Department of Forestry 
and Fire Protection (CDF) [http://www.fs.fed.us/r5/spf/about/ 
fhp-change.shtml]. Their program carries out vegetation 
mapping and long-term monitoring using high spatial 
resolution remotely sensed data. Not only extent, but condition 
and cause in the case of change in land cover, are posted. 

As the availability of frequently acquired satellite data sets 
for Earth science applications is burgeoning, we see an 
opportunity for its utilization. With the near-term advent of the 
National Polar-orbiting Operational Environmental Satellite 
System [NPOESS] and the NPOESS Preparatory Project 
Mission [NPP] even more data will become available. 
Currently, beside archival data, satellite instrument data sets are 
available in real-time through direct broadcast or in near-real- 
time through a number of venues. These data are available for 
analysis within a few minutes, or at most a few hours, from 
acquisition. This immediacy of access calls for immediacy of 
analysis and potential action when used for environmental, 
security, hazards monitoring. In fact many programs are 
promoting global change monitoring strategies that would use 
frequent acquisitions and analysis [9]. This project will focus 
on change monitoring using near-real-time data in a continuous 
mode, not at a yearly, multi-year or decadal scale using 
composite data sets. Due to the potentially large amounts and 
frequency of data, an automated system is critical for its 
success. 

In the next section we describe a collection of MODIS we 
have assembled for our experiments. After that we present 
some preliminary analysis results from processing these 
MODIS data sets with RHSEG and outline our planes for 
additional analysis. 


II. MODIS Data Sets Assembled for this Study 

A. Acquisition of Data Sets 

MODIS data is available from a number of sources, but we 
used the NASA Goddard Space Flight Center Data and 
Information Services Center [DISC] Distributed Active 
Archive Center [DAAC] as our source. Since we were seeking 
multi-date MODIS granules, we use the MODIS Multiple Data 
Ordering Page [MDOP] web interface. [Data Access, MODIS 
Multiple Data through http://daac.gsfc.nasa.gov/MODIS]. The 
interface allows spatial searching with a 'rubber band' on map 
or geographic coordinate box. The site offers a wide range of 
MODIS radiometric, geolocation, atmosphere and oceans 
products. A channel subset function is available for some 1KM 
data sets. Recent data is available through immediate FTP, 
while archived data is staged for push or pull FTP from the 
library and users advised by email of data availability for 
download. 

B. Data Sets Used in this Study 

We used MODIS Terra MOD021KM calibrated at aperture 
radiances and MOD03 geolocation data. From the 
MOD021KM data, we acquired only the reflected bands 1 -7, a 
feature of the MDOP system. Since this was a study of change 
detection, we acquired seasonal data sets from early 2003 
through early 2005. The acquisitions were meant to capture 
both seasonal change, and as a test of change detection, large 
chaparral fires that occurred in Southern California in fall 2003. 
We bounded our search with the geographic coordinate spatial 
search, using 3 2°N, 30° N, 1 1 7°W and 1 16°W as our limits. We 
collected 12 MODIS granules for analysis. Our selection 
criteria included seasonality and cloud cover over the Southern 
California region. Once data orders were made, the data 
granules were usually available over night for FTP pull. Data 
sets are uncompressed, distributed in HDF. 


TABLET MODIS Data sets used 


MOD03 & MOD021KM Data Granules 

Granule name 

Date 

Notes 

A2003031.1815 

31 JAN 2003 

Winter 

A2003 109. 1825 

19 APR 2003 

Spring 

A2003221.1825 

09 AUG 2003 

Summer 

A2003294.1820 

21 OCT 2003 

Pre-fire 

A200330I.1825 

28 OCT 2003 

Fire 

A2003322.1845 

18 NOV 2003 

Post-fire 

A2004032.1825 

01 FEB 2004 

Winter 

A2004080.1825 

20 MAR 2004 

Spring 

A2004163.1855 

1 1 JUN 2004 

Summer 

A2004268.1850 

24 SEP 2004 

Summer 

A2004334.1835 

29 NOV 2004 

Winter 

A2005059.1820 

28 FEB 2005 

Spring 


i. only used granules from Terra, AMI platform 





C. MOD/S Data Processing 

Prior to image segmentation and change analysis the seven 
reflective band MODIS data were converted from swath to 
georectified grid, from HDF to binary BSQ, normalized 
difference vegetation index calculated and added as an eighth 
band, and then a uniform geographic subset made so that all 12 
data sets were coincident. The final data sets were 
approximately 1000 X 1000 km with a 1 km spatial resolution. 
Data were moved to our local network attached storage device 
using a freeware GUI-based Windows FTP client program 
[FileZilla, fileziIla.sourceforge.net]. Further processing was 
carried out with ENVI [the Environment for Visualizing 
Images, Research Systems, Inc., Boulder, CO]. 

III. PRELIMINARY ANALYSIS WITH RHSEG 

A. Land/Water Mask Generation 

We will be primarily interested in monitoring changes in 
land cover. Therefore, an initial analysis task is to generate a 
land/water mask. Another reason for generating a landAvater 
mask is that such a mask makes a cloud mask much easier, 
since clouds can look spectrally different over water than they 
look over land. However, a quick look at all twelve MODIS 
data sets shows that not one data set is completely cloud free. 
This complicates the task of generating a land/water mask. 

The most cloud free data set is A2004334.1835, which is 
displayed in Fig.l. The coarsest segmentation from the 
segmentation hierarchy produced by RHSEG produces a two 
region segmentation in which the two regions correspond 
primarily to water and to land. However, there are a few small 
sub-regions in what should be ocean that are labeled as land, 
due to some small clouds over the ocean (lower center of Fig. 
1). This confusion does not occur over land, since the two 
region segmentation places clouds with land. 

This problem can be corrected by taking advantage of the 
transient nature of clouds, through the combining of the 
analysis of data set A2004334. 1 835 with other relatively cloud- 
free data sets. The next two most cloud-free data sets are 
A2003322.1845 and A2004268.1850, which are displayed in 
Fig. 2 and Fig. 3, respectively. Even though there is a large 
cloud bank in the lower left comer of Fig. 2, and ... of Fig. 3, 
these images are still useful in helping to determine the 
land/water mask since there is at least one image where the 
water areas are cloud-free. 

The land'water mask can now be generated by designating 
as land only those areas that are in the “land” region for all 
three data sets (from the coarse two region segmentations 
produced by RHSEG). The result is shown in Fig. 4. 


Figure 1. MODIS granule A2004334.1835, bands 3, 2 and 1 display as Red , 
Green and Blue. This is the most cloud-free data set of the twelve used in this 
study. 


Figure 2. MODIS granule A2003322. 1835, bands 3, 2 and 1 display as Red , 
Green and Blue. This is one of the three data sets, along with Fig. 1 and Fig.3. 
used to generate a land/water mask. 
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Figure 3. MODIS granule A2004268.1835, bands 3, 2 and 1 display as Red . 
Green and Blue. This is one of the three data sets, along with Fig. 1 and Fig.2, 
used to generate a land/water mask. 



Figure 4. A land/water mask produce from combining the coarsest RHSEG 
segmentations from data sets A2004334.1835, A2003322.1835, and 
A2004268.1835. 

B. Cloud Mask Generation and Detection of other Bright 
Regions ( over land) 

The Land/Water mask was applied to each data set, and 
RHSEG was used to process just the land image pixels. 
Information on selected regions from the segmentation 


hierarchy for each data set is given in TABLE II. The 
segmentation hierarchy was initially examined to see if there 
were any bright regions that were so dominate that they 
persisted all the way to the coarsest level of the segmentation 
hierarchy. This was the case only for MODIS granule 
A2003031.1815, where a 25,873 pixel cloudy area (region 25) 
persisted all the way up to the coarsest hierarchical level (level 
35). Before additional analysis is performed on this data set, it 
would be probably advisable to mask out this dominate cloud 
region. 

For the other data sets, the bright regions (clouds, snow or 
smoke) did not persist as separate regions up to the coarsest 
level of the segmentation hierarchy. As detailed in Table II, 
for MODIS granule A2003 109.1825, a 50,054 pixel region that 
is apparently mountaintop snow persists all the way up to the 
second coarsest level of the segmentation hierarchy. For 
MODIS granule A2003221.1825, a small 7,524 pixel region of 
mountain ridge clouds also persists all the way up to the second 
coarsest level of the segmentation hierarchy. There are no 
bright regions in MODIS granule A2003294.1820. However, 
some cloud and smoke regions appear in MODIS granule 
A2003301.1825. Here region 34, which combines both clouds 
and smoke, persists to hierarchical 23 out of 27. The smoke 
region can be separated out from the cloud region as region 35 
at hierarchical level 8. Two small mountain snow regions are 
seen in MODIS granule A2003322.1845. One is region 52 at 
hierarchical level 29 out of 32 and the other is region 55 at 
hierarchical level 23 out of 32. A region combining clouds and 
snow (region 26) persists all the way up to the second coarsest 
level of the segmentation hierarchy in MODIS granule 
A2004032.1825. This region can be separated into constituent 
regions of snow and clouds at finer levels of the segmentation 
hierarchy (not shown in TABLE II). Three small bright regions 
appear in MODIS granule A2004080.1825. Region 44 at 
hierarchical level 29 out of 30 is mountain snow, region 48 at 
hierarchical level 26 covers cloudy areas along the coast, and 
region 45 at hierarchical level 23 is apparently mountaintop 
clouds. MODIS granule A2004 163. 1855 has two small cloudy 
regions, regions 56 and 55, persisting up to hierarchical levels 
30 and 29, respectively, out of 35 hierarchical levels. Similarly, 
MODIS granule A2004268.1850 has two small regions that are 
apparently clouds, regions 49 and 59, persisting up to 
hierarchical level 23 out of 28 hierarchical levels. A fairly large 
snow region of 14,318 pixels (region 31) persists up to the 
second coarsest hierarchical level in MODIS granule 
A2004334.1835. A smaller region (region 44) that is 
apparently clouds persists up to hierarchical level 26 out of 30. 
Finally, a moderately large cloudy region (region 33) persists 
up to the second coarsest hierarchical level in MODIS granule 
A2004049.1820. 

The above analysis of the RHSEG segmentation hierarchies 
was performed with a user-interactive analysis tool called 
HSEGViewer. This tool is available along with a time-limited 
demo version of the RHSEG program through the NASA 
Goddard Space Flight Center’s Technology Transfer Office at 
http://techtransfer.gsfc.nasa.gov/RHSEG/ . An automated 
approach for identifying bright regions in the segmentation 
hierarchies should be ready for discussion at the workshop. 



IV. Plans for Additional Analysis 

More detailed studies of change will focus on changes at 
the landscape level such as those due to interannual variability 
in vegetation areal extent and status, variability in seasonality 
[onset of growth, drought conditions] or more acute, but short 
lived phenomenon like insect or weather-related damage, or 
longer term impacts like fire. The spectral coverage of MOD1S 
even makes it possible to detect algal blooms, sediment plumes 
and other disturbance in water bodies. Within the two year 
period of the data collected for this project, examples can be 
found of all of these changes and more. 

One of our major goals is to create an automated change 
detection algorithm that can not only flag areas of change for 
‘human’ follow up, but that can determine if the change is 
‘important’ or not. The definition of ‘importance’ would 
depend on the final use of the system, and the operational 
agency and its needs, but filtering is critical so that every 
‘change’ doesn’t create a flag. We envision filtering out 
ephemeral changes due to weather, short-term precipitation 
regimes or recognized seasonality. The use of long-term, but 
high frequency of acquisition data would allow training the 
change detection for effective filtering of potential false 
positives. 
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TABLE II. Selected Regions from the Segmentation Hierarchy for each Data Set. 
(C) designates clouds, (Sn) designates snow, and (Sm) designates smoke. 


Mean Vector 


(5499, 9961, 10703, 9692, 9231, 8571, 7166) 
(2057, 4752, 3544, 3286, 5073, 5262, 5081) 
(4410,9260,6270, 6608, 9601, 9970, 10099) 



E WMlMfrkhn 


(7216, 13391, 12150, 12009, 1 1 739, 9805, 8307) 


(4129, 8682, 5422, 5938, 9530, 10537, 10277 


(2448, 6574, 3972, 3878, 7136, 7230, 6251) 


(9938, 17035, 16531, 16601, 14732, 11293,8696) 


(2705, 6137,3787, 3919, 6758, 7323, 7260) 


(1601, 4313, 2907, 2602, 4760, 4736, 4199) 


(3043, 6458, 3950, 4307, 7070, 7751, 7914) 


(1917, 4583, 3240, 3040, 5013, 5043, 4599 


(2641,5345,4687,4405, 5351, 4883, 4249) 


(2539,4716, 4978, 4547, 4572, 4448, 4058) 


(1268, 3177, 2413, 2091, 3418, 3383, 3000) 


(2104, 4758,3060,3074, 5266, 5768, 5672) 


(6720, 10762, 10806. 11117, 4749, 1633, 1308) 




(3116, 6424,4859, 4828, 6088, 5784, 5536) 


(1577, 3750, 2686, 2461, 4086, 4319, 4189) 


(4365, 8354, 7933, 7388, 7600, 6190, 4493) 


(4329, 8842, 5819, 6268, 9027, 9248, 9580) 


2373, 6039, 3890, 3771, 6091, 6102. 5778) 


(9973, 14467, 16929, 16901,4030, 1362, 1203) 


(8008, 14955, 13442, 13241, 13642, 13274, 1 1868 


(7332, 13058, 12133, 1 1934, 1 1438, 6707, 4862) 


(4430, 9296, 5457, 6018, 9927, 11217, 12042) 


245 1 , 6164, 3818, 3662, 6660, 6964, 6569) 


6933, 13305, 11376, 11191, 12432, 11805, 11084 


(6140, 9656, 9553, 9975, 5226, 4584, 4909) 


(3217, 7058, 4150, 4403, 7611, 8382. 8567) 


(1805, 4683, 2978, 2747, 5053, 5210, 4808) 


(5595, 10482, 9488, 9001, 9633, 6548, 4921 


(6244, 11798, 10277, 10030, 11027, 10279,9595) 
(2706, 5740, 371 1, 3862, 5557, 5842, 6243) 


(1361, 3530, 2435, 2180, 3569, 3492, 3264) 


(5113, 9262, 8352, 8203, 5405, 2239, 1687) 


(3954, 7561, 6455, 6264, 6644, 6025, 6062 


(5465, 1 1014, 9041, 8659, 9708, 8479, 7105) 


(1992, 5715, 3425, 3209, 5538, 5262, 4685) 


(8000, 14658, 13619, 12899, 12160, 9313, 6447) 






























































































